Using simulation studies to evaluate statistical methods

Simulation studies are computer experiments that involve creating data by pseudorandom sampling. The key strength of simulation studies is the ability to understand the behaviour of statistical methods because some 'truth' (usually some parameter/s of interest) is known from the process of generating the data. This allows us to consider properties of methods, such as bias. While widely used, simulation studies are often poorly designed, analysed and reported. This tutorial outlines the rationale for using simulation studies and offers guidance for design, execution, analysis, reporting and presentation. In particular, this tutorial provides: a structured approach for planning and reporting simulation studies, which involves defining aims, data-generating mechanisms, estimands, methods and performance measures ('ADEMP'); coherent terminology for simulation studies; guidance on coding simulation studies; a critical discussion of key performance measures and their estimation; guidance on structuring tabular and graphical presentation of results; and new graphical presentations. With a view to describing recent practice, we review 100 articles taken from Volume 34 of Statistics in Medicine that included at least one simulation study and identify areas for improvement.Comment: 31 pages, 9 figures (2 in appendix), 8 tables (1 in appendix

Assessment and Implication of Prognostic Imbalance in Randomized Controlled Trials with a Binary Outcome – A Simulation Study

Chance imbalance in baseline prognosis of a randomized controlled trial can lead to over or underestimation of treatment effects, particularly in trials with small sample sizes. Our study aimed to (1) evaluate the probability of imbalance in a binary prognostic factor (PF) between two treatment arms, (2) investigate the impact of prognostic imbalance on the estimation of a treatment effect, and (3) examine the effect of sample size (n) in relation to the first two objectives.We simulated data from parallel-group trials evaluating a binary outcome by varying the risk of the outcome, effect of the treatment, power and prevalence of the PF, and n. Logistic regression models with and without adjustment for the PF were compared in terms of bias, standard error, coverage of confidence interval and statistical power.For a PF with a prevalence of 0.5, the probability of a difference in the frequency of the PF≥5% reaches 0.42 with 125/arm. Ignoring a strong PF (relative risk = 5) leads to underestimating the strength of a moderate treatment effect, and the underestimate is independent of n when n is >50/arm. Adjusting for such PF increases statistical power. If the PF is weak (RR = 2), adjustment makes little difference in statistical inference. Conditional on a 5% imbalance of a powerful PF, adjustment reduces the likelihood of large bias. If an absolute measure of imbalance ≥5% is deemed important, including 1000 patients/arm provides sufficient protection against such an imbalance. Two thousand patients/arm may provide an adequate control against large random deviations in treatment effect estimation in the presence of a powerful PF.The probability of prognostic imbalance in small trials can be substantial. Covariate adjustment improves estimation accuracy and statistical power, and hence should be performed when strong PFs are observed

Why Were More Than 200 Subjects Required to Demonstrate the Bioequivalence of a New Formulation of Levothyroxine with an Old One?

At the request of French Regulatory Authorities, a new formulation of Levothyrox® was licensed in France in 2017, with the objective of avoiding the stability deficiencies of an existing licensed formulation. Before launching the new formulation, an average bioequivalence (ABE) trial was conducted, having enrolled 204 subjects and selected for interpretation a narrow a priori bioequivalence range of 0.90–1.11. Bioequivalence was concluded. In a previous publication, we questioned the ability of an ABE trial to guarantee the switchability within patients of the new and old levothyroxine formulations. It was suggested that the two formulations should be compared using the conceptual framework of individual bioequivalence. The present paper is a response to those claiming that, despite the fact that ABE analysis does not formally address the switchability of the two formulations, future patients will nevertheless be fully protected. The basis for this claim is that the ABE study was established in a large trial and analyzed using a stringent a priori acceptance interval of equivalence. These claims are questionable, because the use of a very large number of subjects nullifies the implicit precautionary intention of the European guideline when, for a Narrow Therapeutic Index drug, it recommends shortening the a priori acceptance interval from 0.80–1.25 to 0.90–1.11

A standardized framework for the validation and verification of clinical molecular genetic tests

The validation and verification of laboratory methods and procedures before their use in clinical testing is essential for providing a safe and useful service to clinicians and patients. This paper outlines the principles of validation and verification in the context of clinical human molecular genetic testing. We describe implementation processes, types of tests and their key validation components, and suggest some relevant statistical approaches that can be used by individual laboratories to ensure that tests are conducted to defined standards

Silica-Encapsulated Efficient and Stable Si Quantum Dots with High Biocompatibility

A facile fabrication method to produce biocompatible semiconductor Quantum Dots encapsulated in high quality and thick thermal oxide is presented. The process employs sonication of porous Si/SiO2 structures to produce flakes with dimension in the 50–200 nm range. These flakes show a coral-like SiO2 skeleton with Si nanocrystals embedded in and are suitable for functionalization with other diagnostic or therapeutic agents. Silicon is a biocompatible material, efficiently cleared from the human body. The Photoluminescence emission falls in the transparency window for living tissues and is found to be bright and stable for hours in the aggressive biological environment

Accounting for centre-effects in multicentre trials with a binary outcome - when, why, and how?

BACKGROUND: It is often desirable to account for centre-effects in the analysis of multicentre randomised trials, however it is unclear which analysis methods are best in trials with a binary outcome. METHODS: We compared the performance of four methods of analysis (fixed-effects models, random-effects models, generalised estimating equations (GEE), and Mantel-Haenszel) using a re-analysis of a previously reported randomised trial (MIST2) and a large simulation study. RESULTS: The re-analysis of MIST2 found that fixed-effects and Mantel-Haenszel led to many patients being dropped from the analysis due to over-stratification (up to 69% dropped for Mantel-Haenszel, and up to 33% dropped for fixed-effects). Conversely, random-effects and GEE included all patients in the analysis, however GEE did not reach convergence. Estimated treatment effects and p-values were highly variable across different analysis methods. The simulation study found that most methods of analysis performed well with a small number of centres. With a large number of centres, fixed-effects led to biased estimates and inflated type I error rates in many situations, and Mantel-Haenszel lost power compared to other analysis methods in some situations. Conversely, both random-effects and GEE gave nominal type I error rates and good power across all scenarios, and were usually as good as or better than either fixed-effects or Mantel-Haenszel. However, this was only true for GEEs with non-robust standard errors (SEs); using a robust ‘sandwich’ estimator led to inflated type I error rates across most scenarios. CONCLUSIONS: With a small number of centres, we recommend the use of fixed-effects, random-effects, or GEE with non-robust SEs. Random-effects and GEE with non-robust SEs should be used with a moderate or large number of centres

Levothyrox® new and old formulations: are they switchable for millions of patients?

International audienceIn France, more than 2.5 million patients are currently treated with levothyroxine, mainly as the marketed product Levothyrox ®. In March 2017, at the request of French authorities, a new formulation of Levothyrox ® was licensed, with the objective of avoiding stability deficiencies of the old formulation. Before launching this new formulation, an average bioequivalence trial, based on European Union recommended guidelines, was performed. The implicit rationale was the assumption that the two products, being bioequivalent, would also be switchable, allowing substitution of the new for the old formulation, thus avoiding the need for individual calibration of the dosage regimen of thyroxine, using the thyroid-stimulating hormone level as the endpoint, as required for a new patient on initiating treatment. Despite the fact that both formulations were shown to be bioequivalent, adverse drug reactions were reported in several thousands of patients after taking the new formulation. In this opinion paper, we report that more than 50% of healthy volunteers enrolled in a successful regulatory average bioequivalence trial were actually outside the a priori bioequivalence range. Therefore, we question the ability of an average bioequivalence trial to guarantee the switchability within patients of the new and old levothyroxine formulations. We further propose an analysis of this problem using the conceptual framework of individual bioequivalence. This involves investigating the bioavailability of the two formulations within a subject, by comparing not only the population means (as established by average bioequivalence) but also by assessing two variance terms, namely the within-subject variance and the variance estimating subject-by-formulation interaction. A higher within individual variability for the new formulation would lead to reconsideration of the appropriateness of the new formulation. Alternatively, a possible subject-by-formulation interaction would allow a judgement on the ability, or not, of doctors to manage patients effectively during transition from the old to the new formulation

Adjusting for multiple prognostic factors in the analysis of randomised trials

Background: When multiple prognostic factors are adjusted for in the analysis of a randomised trial, it is unclear (1) whether it is necessary to account for each of the strata, formed by all combinations of the prognostic factors (stratified analysis), when randomisation has been balanced within each stratum (stratified randomisation), or whether adjusting for the main effects alone will suffice, and (2) the best method of adjustment in terms of type I error rate and power, irrespective of the randomisation method. Methods: We used simulation to (1) determine if a stratified analysis is necessary after stratified randomisation, and (2) to compare different methods of adjustment in terms of power and type I error rate. We considered the following methods of analysis: adjusting for covariates in a regression model, adjusting for each stratum using either fixed or random effects, and Mantel-Haenszel or a stratified Cox model depending on outcome. Results: Stratified analysis is required after stratified randomisation to maintain correct type I error rates when (a) there are strong interactions between prognostic factors, and (b) there are approximately equal number of patients in each stratum. However, simulations based on real trial data found that type I error rates were unaffected by the method of analysis (stratified vs unstratified), indicating these conditions were not met in real datasets. Comparison of different analysis methods found that with small sample sizes and a binary or time-to-event outcome, most analysis methods lead to either inflated type I error rates or a reduction in power; the lone exception was a stratified analysis using random effects for strata, which gave nominal type I error rates and adequate power. Conclusions: It is unlikely that a stratified analysis is necessary after stratified randomisation except in extreme scenarios. Therefore, the method of analysis (accounting for the strata, or adjusting only for the covariates) will not generally need to depend on the method of randomisation used. Most methods of analysis work well with large sample sizes, however treating strata as random effects should be the analysis method of choice with binary or time-to-event outcomes and a small sample size

Reporting on covariate adjustment in randomised controlled trials before and after revision of the 2001 CONSORT statement: a literature review

<p>Abstract</p> <p>Objectives</p> <p>To evaluate the use and reporting of adjusted analysis in randomised controlled trials (RCTs) and compare the quality of reporting before and after the revision of the CONSORT Statement in 2001.</p> <p>Design</p> <p>Comparison of two cross sectional samples of published articles.</p> <p>Data Sources</p> <p>Journal articles indexed on PubMed in December 2000 and December 2006.</p> <p>Study Selection</p> <p>Parallel group RCTs with a full publication carried out in humans and published in English</p> <p>Main outcome measures</p> <p>Proportion of articles reported adjusted analysis; use of adjusted analysis; the reason for adjustment; the method of adjustment and the reporting of adjusted analysis results in the main text and abstract.</p> <p>Results</p> <p>In both cohorts, 25% of studies reported adjusted analysis (84/355 in 2000 vs 113/422 in 2006). Compared with articles reporting only unadjusted analyses, articles that reported adjusted analyses were more likely to specify primary outcomes, involve multiple centers, perform stratified randomization, be published in general medical journals, and recruit larger sample sizes. In both years a minority of articles explained why and how covariates were selected for adjustment (20% to 30%). Almost all articles specified the statistical methods used for adjustment (99% in 2000 vs 100% in 2006) but only 5% and 10%, respectively, reported both adjusted and unadjusted results as recommended in the CONSORT guidelines.</p> <p>Conclusion</p> <p>There was no evidence of change in the reporting of adjusted analysis results five years after the revision of the CONSORT Statement and only a few articles adhered fully to the CONSORT recommendations.</p